DSS: Data Stream Scan

dss (data stream scan) is a framework for describing, transforming, reading, querying, and writing streams of record oriented data. dss is implemented as a command and library API and used extensively to aid network measurements in our organization. The API is extended by DLLs (shared libraries) that define data domain specific I/O, type and query functions. We provide a best-in-class repository for data scanning, along with up-to-date documentation as a side effect of coding to the API. Many large scale network applications have used dss significantly reducing the time spent in coding and in querying data. The reasons for the success of dss are its lightweight and extensible architecture, generic usage template, scope of supported data domains, scalability, and speed. dss compares extremely favorably against perl, the typical recourse in the networking community, and against customized C/C++ code written to deal with specific data sets. dss has been successfully applied over three years over a wide range of applications, including large volumes of NETFLOW data, BGP tables, HTTP proxy and server logs, logs, and OSPF LSA (Link State Advertisements.) The use of dss has led to fearless exploration of ideas across very large volumes of data.